1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a computer-readable storage medium on which a program for executing image processing is stored, and specifically, to processing for extracting a particular object from frames of moving image data.
2. Related Background Art
In recent years, coding systems have been studied in which a moving picture is grasped as a synthesis of components called objects, and in which compression coding is performed on an object basis. Standardization of such systems known as Moving Picture Experts Group Phase 4 (MPEG-4) is being advanced. A moving picture object can have any shape and is expressed by a combination of data called xe2x80x9cshapexe2x80x9d representing information on the shape and data called xe2x80x9ctexturexe2x80x9d representing the contents of the image.
Known object generation methods include a chroma key separation method using a studio set, etc., a method for generating a desired object by computer graphics (CG), and a method of extracting an object from a natural image.
The chroma key separation method adopts a process in which a uniform blue background called a blue back is prepared in a studio, an image including the blue back is photographed, and a desired object is extracted from the photographed image by removing the blue portion.
An essential feature of computer graphics resides in freely generating an image in any shape and, therefore, it is not necessary to pay a special attention to an extraction processing when computer graphics is used. Animation images can be processed in the same manner as CG images if each of cell images (images on celluloid sheets) is regarded as an object.
As a means for extracting an object from a natural image, the method of minimizing energy of an active contour model called xe2x80x9cSnakesxe2x80x9d (see, for example, Michael Kass, Andrew Witkin, and Demetri Terzopoulos, xe2x80x9cSnakes: Active Contour Modelsxe2x80x9d, International Journal of Computer Vision, Vol. 1, No. 3, pp. 321-331, 1988) is well known.
In snakes representation, an energy function which minimizes when a contour is extracted is defined and its minimal solution is obtained by iterative calculation from a suitable initial value. The energy function is defined as the linear sum of external energy provided as constraints on passing through edge points and internal energy provided as constraints on smoothness.
To use a snake, it is necessary to roughly designate a contour of an extraction target as an initial contour. In processing a moving picture, setting of an initial contour is required with respect to each frame. However, automatization of processing can be achieved by setting the results of extraction from the previous frame as initial values in the current frame. A technique for obtaining corresponding contour lines between frames is called tracking.
Each of the above-described extraction methods entails a problem difficult to solve. That is, the chroma key method requires a uniform background color and, hence, a large-scale studio set if a high extraction accuracy is specified. Also, if an object extraction target contains the background color, the target cannot be correctly extracted. Therefore the selection of colors of targets is restricted.
Computer graphics and animation can be used without extraction processing but have a fatal drawback of being incapable of processing natural images such as those in pictures taken by video cameras.
The methods of extracting objects from natural images have the advantage of reducing restrictions on image contents and having high versatility but have the drawback of requiring designation of initial contours with certain accuracy. This is because a minimal solution of an active contour model is largely influenced by an initial contour. That is, if an initial contour differs largely from the actual contour of an object, the convergent result of calculation does not match the actual contour of the object. Ordinarily, initial contour setting is made by a user using a graphical user interface (GUI) including a mouse or the like. The operability of the process of manually setting an initial contour is low and it is difficult to manually make a sufficiently reproducible initial contour setting. The burden on the user in manually setting an initial contour becomes larger as the shape of the contour becomes more complicated.
As a solution to this problem, a method for automatically generating an initial contour may be used. Use of information on movements between frames as well as information on colors and textures in frames is effective in automatically extracting an initial contour. To use such movement information, however, it is necessary that relative movements between the extraction target and the background be detected.
This problem will be described in more detail with reference to FIG. 1. FIG. 1 is a diagram schematically showing movements of objects between frames in a moving picture sequence. Frames F1 to F6 appear successively with passage of time. Frame F1 is a start frame and frame F6 is an end frame. Start frame F1 and the next frame F2 have completely the same internal configurations with no movement of the subject or the background. In this case, since the background and the object are equal in movement, the object cannot be extracted even if processing for obtaining the amount of movement between the frames is performed. Processing ends with the same result in a case where the camera is panned when both the background and the object are at rest.
If a frame remote from the start frame with respect to time, e.g., frame F4 is referred to instead of the adjacent frame, the possibility of success in detecting a relative movement between the object and the background is increased. In such a case, however, it is difficult to recognize the correspondence between the images of the background and between the images of the object if the search range is not increased. This is because, if the time interval is increased, the possibility of increase of a change in shape is high. Consequently, the method of setting an initial contour by obtaining a movement from the start frame is rather low in versatility and entails difficulty in obtaining good results in subsequent tracking. Also, the method of setting an initial contour by obtaining a movement from the start frame does not always ensure good extraction results.
According to the above-mentioned background, an object of the present invention is to provide an image processing apparatus, which facilitates user operation and can extract objects more correctly from frames of moving image data, and an image processing method and a computer-readable storage medium on which a program for executing image processing is stored.
An image processing method according to a preferred embodiment of the present invention is characterized in that the method comprises: a selection step of selecting a particular frame from a plurality of frames of moving image data; a generation step of generating an initial contour corresponding to an extraction target in the frame selected in the selection step; a convergence step of converging the initial contour to the contour of the extraction target; and a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of convergence in the convergence step, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
Further, according to an another preferred embodiment of the present invention, there is provided an image processing method comprising: a generation step of generating an initial contour in each of a plurality of frames of moving image data; a selection step of selecting the initial contour in specific one of the frames from the initial contours generated in the frames in the generation step; a convergence step of converging the initial contour in the selected frame to the contour of an extraction target; and a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of convergence in the convergence step, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
Further, according to an another preferred embodiment of the present invention, there is provided an image processing method comprises: a generation step of generating an initial contour in each of a plurality of frames of moving image data; a convergence step of converging the initial contour in each frame to the contour of an extraction target; a selection step of selecting a specific frame from the contours in the frames converged in the convergence step; and a control step of setting an initial contour corresponding to the extraction target in an other frames on the basis of the result of contour convergence in the selected frame, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still preferred embodiment of the present invention, there is provided an image processing method comprising: a generation step of generating an initial contour in each of a plurality of frames of moving image data; a convergence step of converging the initial contour in each frame to the contour of an extraction target; a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of contour convergence in each frame, and effecting convergence to the contour of the extraction target on the basis of the set initial contour; and a selection step of selecting a specific result from the convergence results relating to the other frame on the basis of the result of contour convergence in each frame, wherein frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still another preferred embodiment of the present invention, there is provided an image processing method comprises: an input step of inputting a plurality of frames of moving image data; a selection step of selecting a specific one of the plurality of frames; and an extraction step of extracting image data of a desired object from the frames in the direction from the specific frame to a frame preceding the specific frame with respect to time and in the direction from the specific frame to a frame subsequent to the specific frame with respect to time.
According to a still another preferred embodiment of the present invention, there is provided an image processing apparatus comprises: selection means for selecting a specific frame from a plurality of frames of moving image data; generation means for generating an initial contour corresponding to an extraction target in the frame selected by the selection means; convergence means for converging the initial contour to the contour of the extraction target; and control means for setting an initial contour corresponding to the extraction target in an other frame on the basis of an output from the convergence means, and for effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein the control means can set frames preceding and subsequent to the selected frame with respect to time as the other frame.
According to a still another preferred embodiment of the present invention, there is provided an image processing apparatus comprises: generation means for generating an initial contour in each of a plurality of frames of moving image data; selection means for selecting the initial contour in specific one of the frames from the initial contours generated in the frames by the generation means; convergence means for converging the initial contour in the selected frame to the contour of an extraction target; and control means for setting an initial contour corresponding to the extraction target in an other frame on the basis of an output from the convergence means, and for effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein the control means can set frames preceding and subsequent to the selected frame with respect to time as the other frame.
According to a still another preferred embodiment of the present invention, there is provided an image processing apparatus comprises: generation means for generating an initial contour in each of a plurality of frames of moving image data; convergence means for converging the initial contour in each frame to the contour of an extraction target; selection means for selecting a specific frame from the contours in the frames converged by the convergence means; and control means for setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of contour convergence in the selected frame, and for effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein the control means can set frames preceding and subsequent to the selected frame with respect to time as the other frames.
According to a still another preferred embodiment of the present invention, there is provided an image processing apparatus comprises: generation means for generating an initial contour in an a plurality of frames of moving image data; convergence means for converging the initial contour in each frame to the contour of an extraction target; control means for setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of contour convergence in each frame, and for effecting convergence to the contour of the extraction target on the basis of the set initial contour; and selection means for selecting a specific result from the convergence results relating to the other frame on the basis of the result of contour convergence in each frame, wherein the control means is capable of setting frames preceding and subsequent to the selected frame with respect to time as the other frames.
According to a still another preferred embodiment of the present invention, there is provided an image processing apparatus comprises: input means for inputting a plurality of frames of moving image data; selection means for selecting specific one of the plurality of frames; and extraction means for extracting image data of a desired object from frames in the direction from the specific frame to a frame preceding the specific frame with respect to time and in the direction from the specific frame to a frame subsequent to the specific frame with respect to time.
Further, according to an another preferred embodiment of the present invention, there is provided a computer-readable storage medium stores: a code for a selection step of selecting a specific frame from a plurality of frames of moving image data; a code for a generation step of generating an initial contour corresponding to an extraction target in the frame selected in the selection step; a code for a convergence step of converging the initial contour to the contour of the extraction target; and a code for a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of convergence in the convergence step, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still another preferred embodiment of the present invention, there is provided a computer-readable storage medium stores: a code for a generation step of generating an initial contour in an a plurality of frames of moving image data; a code for a selection step of selecting the initial contour in specific one of the frames from the initial contours generated in the frames in the generation step; a code for a convergence step of converging the initial contour in the selected frame to the contour of an extraction target; and a code for a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of convergence in the convergence step, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still another preferred embodiment of the present invention, there is provided a computer-readable storage medium stores: a code for a generation step of generating an initial contour in an a plurality of frames of moving image data; a code for a convergence step of converging the initial contour in each frame to the contour of an extraction target; a code for a selection step of selecting a specific frame from the contours in the frames converged in the convergence step; and a code for a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of contour convergence in the selected frame, and effecting convergence to the contour of the extraction target on the basis of the set initial contour, wherein, in the control step, frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still another preferred embodiment of the present invention, there is provided a computer-readable storage medium stores: a code for a generation step of generating an initial contour in an a plurality of frames of moving image data; a code for a convergence step of converging the initial contour in each frame to the contour of an extraction target; a code for a control step of setting an initial contour corresponding to the extraction target in an other frame on the basis of the result of contour convergence in each frame, and effecting convergence to the contour of the extraction target on the basis of the set initial contour; and a code for a selection step of selecting a specific result from the convergence results relating to the other frame on the basis of the result of contour convergence in each frame, wherein frames preceding and subsequent to the selected frame with respect to time can be set as the other frame.
According to a still another preferred embodiment of the present invention, there is provided computer-readable storage medium stores: a code for an input step of inputting a plurality of frames of moving image data; a code for a selection step of selecting specific one of the plurality of frames; and a code for an extraction step of extracting image data of a desired object from frames in the direction from the specific frame to a frame preceding the specific frame with respect to time and in the direction from the specific frame to a frame subsequent to the specific frame with respect to time.
Other objects, features and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.